Techniques for sharing mapping data between an unmanned aerial vehicle and a ground vehicle

ABSTRACT

Techniques are disclosed for sharing sensor information between multiple vehicles. A system for sharing sensor information between multiple vehicles, can include an aerial vehicle including a first computing device and first scanning sensor, and a ground vehicle including a second computing device and second scanning. The aerial vehicle can use the first scanning sensor to obtain first scanning data and transmit the first scanning data to the second computing device. The ground vehicle can receive the first scanning data from the first computing device, obtain second scanning data from the second scanning sensor, identify an overlapping portion of the first scanning data and the second scanning data based on at least one reference object in the scanning data, and execute a navigation control command based on one or more roadway objects identified in the overlapping portion of the first scanning data and the second scanning data.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication PCT/CN2019/077505, filed Mar. 8, 2019, entitled, “TECHNIQUESFOR SHARING MAPPING DATA BETWEEN AN UNMANNED AERIAL VEHICLE AND A GROUNDVEHICLE,” which is related to International ApplicationPCT/CN2019/077504, filed Mar. 8, 2019, entitled, “TECHNIQUES FORCOLLABORATIVE MAP CONSTRUCTION BETWEEN AN UNMANNED AERIAL VEHICLE AND AGROUND VEHICLE” each of which are herein incorporated by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE INVENTION

The disclosed embodiments relate generally to techniques for mapping andobject detection and, more particularly, but not exclusively, totechniques for sharing mapping data between an unmanned aerial vehicleand a ground vehicle.

BACKGROUND

Movable objects, including aerial vehicles such as unmanned aerialvehicles (UAVs) and ground vehicles such as autonomous vehicles anddriven vehicles, can be used for performing surveillance,reconnaissance, and exploration tasks for various applications. Movableobjects may include a payload, including various sensors, which enablesthe movable object to capture sensor data during movement of the movableobjects. The captured sensor data may be viewed on a client device, suchas a client device in communication with the movable object via a remotecontrol, remote server, or other computing device. The sensor data thatcan be captured by the sensors may be limited (e.g., in field of view,view angle, etc.) due to how the sensors are mounted to the movableobject and the positioning of the movable object. Although perspectivetransformations may be used to change the perspective of the sensordata, such transformations can be processing intensive and introducedistortions into the transformed data. This limits the reliability ofsuch transformed data when used in various applications.

SUMMARY

Techniques are disclosed for sharing sensor information between multiplevehicles. A system for sharing sensor information between multiplevehicles, can include an aerial vehicle including a first computingdevice and first scanning sensor, and a ground vehicle including asecond computing device and second scanning. The aerial vehicle can usethe first scanning sensor to obtain first scanning data and transmit thefirst scanning data to the second computing device. The ground vehiclecan receive the first scanning data from the first computing device,obtain second scanning data from the second scanning sensor, identify anoverlapping portion of the first scanning data and the second scanningdata based on at least one reference object in the scanning data, andexecute a navigation control command based on one or more roadwayobjects identified in the overlapping portion of the first scanning dataand the second scanning data.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of an aerial vehicle and a ground vehicle,in accordance with various embodiments.

FIGS. 2A-2C illustrate an example of scanning data of a roadwayenvironment obtained from an aerial vehicle and a ground vehicle, inaccordance with various embodiments.

FIG. 3 illustrates an example of a scanning manager and a detectionmanager, in accordance with various embodiments.

FIG. 4 illustrates an example of a machine learning model for use inroadway object detection, in accordance with various embodiments.

FIG. 5 shows a flowchart of a method of sharing sensor informationbetween multiple vehicles in a movable object environment, in accordancewith various embodiments.

FIG. 6 illustrates an example of an aerial vehicle and a ground vehicle,in accordance with various embodiments.

FIG. 7 illustrates an example of generating a map of a movable objectenvironment using an aerial vehicle and a ground vehicle, in accordancewith various embodiments.

FIG. 8 illustrates an alternative example of generating a map of amovable object environment using an aerial vehicle and a ground vehicle,in accordance with various embodiments.

FIG. 9 illustrates an example of collaborative map construction by anaerial vehicle mapping manager and a ground vehicle mapping manager, inaccordance with various embodiments.

FIG. 10 shows a flowchart of a method of collaborative map constructionin a movable object environment, in accordance with various embodiments.

FIG. 11 illustrates an example of supporting a movable object interfacein a software development environment, in accordance with variousembodiments.

FIG. 12 illustrates an example of an unmanned aircraft interface, inaccordance with various embodiments.

FIG. 13 illustrates an example of components for an unmanned aircraft ina software development kit (SDK), in accordance with variousembodiments.

DETAILED DESCRIPTION

The invention is illustrated, by way of example and not by way oflimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

The following description of the invention describes target mappingusing a movable object. For simplicity of explanation, an unmannedaerial vehicle (UAV) is generally used as example of a movable object.It will be apparent to those skilled in the art that other types ofmovable objects can be used without limitation.

Self-driving techniques may include various sensing, determining andexecuting tasks, such as environment sensing, path planning, behaviordetermination and control execution. With respect to sensing, aself-driving vehicle can analyze its surrounding environment based ondata gathered by one or more sensors mounted on the vehicle, including,e.g., visual sensors, LiDAR sensors, millimeter wave radar sensors,ultrasound sensors, etc. The sensor data can be analyzed using imageprocessing tools, machine learning techniques, etc. to determine depthinformation and semantic information, to assist the vehicle inidentifying surrounding people and objects. Additionally, LiDAR sensorsprovide accurate, longer distance measurements and positioning data forthe vehicle.

Because the sensors are mounted to the vehicle, there are field of viewand view angle limitations for the sensors. For example, when analyzingdriving lanes, image data may be captured by a front-mounted camera, andthe perspective of this image data may be transformed by projecting thefront-view perspective of the image data to a birds-eye view image(e.g., an overhead perspective). This projection introduces distortionsthat lead to a loss of accuracy in the image data. Perspective effectslead to lane lines, and other object represented in the image data, toconverge the farther they are from the imaging sensor. As such, there isa limit to the length of the environment in front of the vehicle thatcan be clearly recognized by a front mounted camera (or other imagingsensor). Therefore, the lane markings and other objects that are faraway will normally be relatively blurry after the perspectivetransformation. Depending on the type of projection used to obtain thebirds-eye view of the image data, the portions of the image representingobjects farther away from the imaging sensor can become more distorted,making it difficult to reliably apply image processing techniques, suchas Canny edge detection, binary image analysis, and other techniques toidentify lane markings and other objects in the image data.

Additionally, the perspective transformation operation requiresparticular cameras to be effective, and even so the images must beprepared before they can be transformed. Further, the way the camera isinstalled on the vehicle and the current roadway conditions, such as theangle of the roadway, will have a significant effect on the reliabilityof the transformation and any analysis based on the transformed images.Also, techniques such as those used to obtain binary images requiregradient and color threshold values that may not be generally applicableacross most roadway conditions (e.g., weather conditions, roads thathave fallen into disrepair, and other roadway conditions may reduce thevisibility of some roadway objects, such as lane markings). All of thisanalysis also needs to be able to be performed quickly, howevertraditional techniques can process approximately 4.5 frames per second(FPS), while the number of frames of a vehicle mounted camera may be 30FPS or above.

In some embodiments, as a car travels on a roadway it may use cameras tocapture images of the roadway environment. These images may includerepresentations of other nearby cars, trees, light poles, signs, andother nearby objects. In prior systems, these images may be transformedfrom a forward perspective view to an overhead view, which may then beused to generate a local map. However, because of how the camera mountedto the front of the car, the camera has a limited field of view. Whenthese images are transformed to an overhead view, the transformationintroduces inaccuracies, such as blurriness or other distortions. Also,the transformation itself requires additional time and processingresources, as discussed above. Due to the inaccuracies in thetransformed images, the resulting maps generated based on these imagesare also less accurate and less useful. This also limits the usefulnessand/or reliability of features that rely on these maps, such as lanedetection and other assisted driving functions. Instead of relying onlyon transformed images, embodiments may use images captured by a drone,or other unmanned aerial vehicle (UAV), which can capture overheadimages directly, without any transformation or the associatedinaccuracies. The car can then generate a map using its own images andthose collected by the drone, reducing or removing the potentialinaccuracies introduced by transformation. Likewise, the map can begenerated more efficiently without requiring the time or resourcesneeded to transform the images.

Embodiments improve data collection and processing of environmental databy collecting data using an aerial vehicle and transmitting the data toa ground vehicle for use in object detection and other navigation tasks.For example, an aerial vehicle can capture a birds-eye view of theroadway environment and provide it to the ground vehicle for processing.Because the birds-eye view is obtained directly from aerial vehicle, noperspective transformation is required, and therefore the data does nothave the distortions caused by perspective transformation. In variousembodiments, a ground vehicle can use the scanning data (e.g., imagedata, point cloud data, etc.) captured by the aerial vehicle to performautonomous driving and/or assisted driving functions. For example, theground vehicle can perform lane detection using the scanning datareceived from the aerial vehicle. Lane detection may be performed as isgenerally known, such as through using, e.g., Canny edge detection toidentify lane markings in the scanning data that indicate laneboundaries. Additionally, or alternatively, binary image analysis may beperformed. A binary image can be obtained by utilizing gradient andcolor threshold values. Each pixel in the scanning data can be analyzedand, when the value of pixel is higher than the threshold value, thatpixel can be set as 1 and when the pixel is lower than the thresholdvalue, that pixel can be set to zero. A sliding window may be run on theresulting binary image, to calculate a polynomial curve to fit the lanerepresented in the binary image. Because the scanning data received fromthe aerial vehicle has not been transformed, and is therefore notdistorted due to a perspective transformation, the lane detectionperformed on this scanning data provides more accurate lane detectionresults. Additionally, because the perspective transformation is notrequired, the image data can be processed more quickly.

FIG. 1 illustrates an example of a movable object in a movable objectenvironment 100, in accordance with various embodiments. As shown inFIG. 1, client device 110 in a movable object environment 100 cancommunicate with a movable object 104 via a communication link 106. Themovable object 104 can be an unmanned aircraft, an unmanned vehicle, ahandheld device, and/or a robot. The client device 110 can be a portablepersonal computing device, a smart phone, a remote control, a wearablecomputer, a virtual reality/augmented reality system, a personalcomputer, and/or a computing device included in another vehicle (such asa ground vehicle 111, e.g., an autonomous vehicle, an unmanned vehicle,a manned vehicle, etc.). In some embodiments, the client device may beremovable from the ground vehicle 111. For example, the client devicemay be a portable personal computing device (such as a smart phone,tablet, or other portable device), the user may attach the portablepersonal computing device to the ground vehicle via a wired or wirelesscommunication interface, such as USB, Bluetooth, WiFi, or otherinterface. Additionally, the ground device 111 can include acommunication system 120A, which is responsible for handling thecommunication between the client device 110 and the movable object 104via communication system 120B. For example, an unmanned aircraft caninclude uplink and downlink. The uplink can be used for transmittingcontrol signals, the down link can be used for transmitting media orvideo stream.

In accordance with various embodiments, the communication link 106 canbe (part of) a network, which is based on various wireless technologies,such as the WiFi, Bluetooth, 3G/4G/5G, and other radio frequencytechnologies. Furthermore, the communication link 106 can be based onother computer network technologies, such as the internet technology, orany other wired or wireless networking technology. In some embodiments,the communication link 106 may be a non-network technology, includingdirect point-to-point connections such as universal serial bus (USB) oruniversal asynchronous receiver-transmitter (UART).

In various embodiments, movable object 104 in a movable objectenvironment 100 can include a carrier 122 and a payload 123, such asscanning sensor 124. In some instances, the payload 123 may be providedon the movable object 104 without requiring the carrier. In variousembodiments, the payload 123 may include one or more high bandwidthcomponents, such as scanning sensor 124, one or more cameras, a stereovision module, or other payload. The computing device 112 can connect tothe flight controller 114 via UART and/or USB to send and receive datato and from the client device via communication system 120B. In variousembodiments, the computing device 112 may include one or more CPUs,GPUs, field programmable gate arrays (FPGA), systems on chip (SoC), orother processor(s). In some embodiments, flight controller 114 andcomputing device 112 can be implemented as separate devices (e.g.,separate processors on separate circuit boards). Alternatively, one ormore of the flight controller 114 and computing device 112 can beimplemented as a single device, such as an SoC. In various embodiments,computing device 112 may be removable from the movable object.

Although the movable object 104 is described generally as an aircraft,this is not intended to be limiting, and any suitable type of movableobject can be used. One of skill in the art would appreciate that any ofthe embodiments described herein in the context of aircraft systems canbe applied to any suitable movable object (e.g., a UAV). As used herein,“aerial vehicle” may be used to refer to a subset of movable objectscapable of flight (e.g., aircraft, UAVs, etc.), while “ground vehicle”may be used to refer to a subset of movable objects that travel on theground (e.g., cars and trucks, both manually controlled by a driver andautonomously controlled).

In accordance with various embodiments, the movable object 104 mayinclude one or more movement mechanisms 116 (e.g. propulsionmechanisms), a sensing system 118, and a communication system 120B. Themovement mechanisms 116 can include one or more of rotors, propellers,blades, engines, motors, wheels, axles, magnets, nozzles, animals, orhuman beings. For example, the movable object may have one or morepropulsion mechanisms. The movement mechanisms may all be of the sametype. Alternatively, the movement mechanisms can be different types ofmovement mechanisms. The movement mechanisms 116 can be mounted on themovable object 104 (or vice-versa), using any suitable means such as asupport element (e.g., a drive shaft). The movement mechanisms 116 canbe mounted on any suitable portion of the movable object 104, such onthe top, bottom, front, back, sides, or suitable combinations thereof.

In some embodiments, the movement mechanisms 116 can enable the movableobject 104 to take off vertically from a surface or land vertically on asurface without requiring any horizontal movement of the movable object104 (e.g., without traveling down a runway). Optionally, the movementmechanisms 116 can be operable to permit the movable object 104 to hoverin the air at a specified position and/or orientation. One or more ofthe movement mechanisms 116 may be controlled independently of the othermovement mechanisms, for example by an application executing on clientdevice 110, computing device 112, or other computing device incommunication with the movement mechanisms.

Alternatively, the movement mechanisms 116 can be configured to becontrolled simultaneously. For example, the movable object 104 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 104. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally oriented rotors mayspin in a counterclockwise direction. For example, the number ofclockwise rotors may be equal to the number of counterclockwise rotors.The rotation rate of each of the horizontally oriented rotors can bevaried independently in order to control the lift and/or thrust producedby each rotor, and thereby adjust the spatial disposition, velocity,and/or acceleration of the movable object 104 (e.g., with respect to upto three degrees of translation and up to three degrees of rotation). Asdiscussed further herein, a controller, such as flight controller 114,can send movement commands to the movement mechanisms 116 to control themovement of movable object 104. These movement commands may be based onand/or derived from instructions received from client device 110,computing device 112, or other entity.

The sensing system 118 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 104 (e.g., with respect to various degrees of translation andvarious degrees of rotation). The one or more sensors can include any ofthe sensors, including GPS sensors, motion sensors, inertial sensors,proximity sensors, or imaging sensors (such as cameras). The sensingdata provided by the sensing system 118 can be used to control thespatial disposition, velocity, and/or orientation of the movable object104 (e.g., using a suitable processing unit and/or control module).Alternatively, the sensing system 118 can be used to provide dataregarding the environment surrounding the movable object, such asweather conditions, proximity to potential obstacles, location ofgeographical features, location of manmade structures, and the like.

Flight controller 114 can connect to various functional modules 108,such as a magnetometer, barometer, and inertial measurement unit (IMU).In some embodiments, communication system 120B can connect to computingdevice 112 instead of, or in addition to, flight controller 114. In someembodiments, sensor data collected by the one or more functional modules108 and the sensing system 118 can be passed from the flight controllerto the computing device 112. In some embodiments, the computing device112 can connect to the scanning sensor 124 via a high bandwidthconnection, such as ethernet or universal serial bus (USB). In someembodiments, sensing system may include a positioning sensor. Thepositioning sensor may include a radio, such as a 4G, 5G, or othercellular or mobile network radio. The radio may be used by a real-timekinematic (RTK) module to enhance positioning data collected by a globalnavigation satellite service (GNSS) module, such as GPS, GLOSNASS,Galileo, BeiDou, etc. The RTK module can receive a reference signal froma reference station using the radio and provide a correction to thepositioning data provided by the GNSS module.

As shown in FIG. 1, ground vehicle 111 may also include one or moresensors coupled to client device 110. In some embodiments, these sensorsmay include a scanning sensor 130 and a scanning sensor 132. Scanningsensors 130 and 132 may include a LiDAR sensor, high definition imagingsensor (e.g., a still or video camera), or other sensor that provideshigh resolution scanning of a target environment. Scanning sensor 130may be coupled to an upper position on ground vehicle 111, as shown inFIG. 1. Scanning sensor 130 is positioned to capture scanning data ofthe environment around the ground vehicle. For example, a LiDAR sensorcan be configured to collect point cloud data representing a 360-degreeview of the ambient environment of the vehicle. Similarly, a highdefinition imaging sensor can collect image data (e.g., still images andvideo) of the environment around the ground vehicle. Scanning sensor 132may be coupled to the ground vehicle 111 at a forward position tocapture scanning data of the environment directly in front of the groundvehicle. For example, scanning sensor 132 may collect scanning datarelated to the roadway environment in which the ground vehicle isoperating (e.g., identify roadway objects (such as lane markings, othervehicles, trees and other objects present in the roadway environment),driving conditions (such as weather conditions), traffic information(including information related to nearby vehicles), roadway information(e.g., road type, surface condition, etc.)). In some embodiments, groundvehicle 111 may also include a positioning sensor 134. In someembodiments, the positioning sensor may include a global navigationsatellite service (GNSS) module, such as GPS, GLOSNASS, Galileo, BeiDou,etc. The positioning sensor 134 may include a radio, such as a 4G, 5G,or other cellular or mobile network radio. The radio may be used by areal-time kinematic (RTK) module to enhance positioning data collectedby the GNSS module. The RTK module can receive a reference signal from areference station using the radio and provide a correction to thepositioning data provided by the GNSS module.

The communication system 120B enables communication with client device110 via communication link 106, which may include various wired and/orwireless technologies as discussed above, and communication system 120A.The communication system 120A or 120B may include any number oftransmitters, receivers, and/or transceivers suitable for wirelesscommunication. The communication may be one-way communication, such thatdata can be transmitted in only one direction. For example, one-waycommunication may involve only the movable object 104 transmitting datato the client device 110, or vice-versa. The data may be transmittedfrom one or more transmitters of the communication system 120A of theclient device to one or more receivers of the communication system 120Bof the movable object, or vice-versa. Alternatively, the communicationmay be two-way communication, such that data can be transmitted in bothdirections between the movable object 104 and the client device 110. Thetwo-way communication can involve transmitting data from one or moretransmitters of the communication system 120B to one or more receiversof the communication system 120A of the client device 110, andvice-versa.

In some embodiments, a client device 110 may communicate with a scanningmanager 126 installed on computing device 112 over a transparenttransmission channel of a communication link 106. The transparenttransmission channel can be provided through the flight controller ofthe movable object which allows the data to pass through unchanged(e.g., “transparent”) to the scanning manager or other application oncomputing device 112. In some embodiments, scanning manager 126 mayutilize a software development kit (SDK), application programminginterfaces (APIs), or other interfaces made available by the movableobject, computing device, scanning sensor 124, etc. In variousembodiments, the scanning manager may be implemented by one or moreprocessors on movable object 104 (e.g., flight controller 114 or otherprocessors), computing device 112, remote controller 111, client device110, or other computing device in communication with movable object 104.In some embodiments, scanning manager 126 may be implemented as anapplication executing on client device 110, computing device 112, orother computing device in communication with movable object 104.

In some embodiments, an application executing on client device 110 orcomputing device 112 (such as detection manager 128 or scanning manager126) can provide control data to one or more of the movable object 104,carrier 122, and payload 124 and receive information from one or more ofthe movable object 104, carrier 122, and payload 124 (e.g., positionand/or motion information of the movable object, carrier or payload;data sensed by the payload such as image data captured by a payloadcamera; and data generated from image data captured by the payloadcamera). In some embodiments, the application may base the control dataon scanning data received from scanning sensor 130 and/or scanningsensor 132.

In some embodiments, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the movement mechanisms 116), and/or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 122).The control data from the application may result in control of thepayload, such as control of the operation of scanning sensor 124, acamera or other image capturing device (e.g., taking still or movingpictures, zooming in or out, turning on or off, switching imaging modes,change image resolution, changing focus, changing depth of field,changing exposure time, changing viewing angle or field of view).

In some instances, the communications from the movable object, carrierand/or payload may include information from one or more sensors (e.g.,of the sensing system 118 or of the scanning sensor 124 or otherpayload) and/or data generated based on the sensing information. Thecommunications may include sensed information from one or more differenttypes of sensors (e.g., GPS sensors, motion sensors, inertial sensor,proximity sensors, or imaging sensors). Such information may pertain tothe position (e.g., location, orientation), movement, or acceleration ofthe movable object, carrier, and/or payload. Such information from apayload may include data captured by the payload or a sensed state ofthe payload.

In some embodiments, computing device 112 can be added to the movableobject. The computing device can be powered by the movable object andcan include one or more processors, such as CPUs, GPUs, fieldprogrammable gate arrays (FPGAs), system on chip (SoC),application-specific integrated circuit (ASIC), or other processors. Thecomputing device can include an operating system (OS), such as Windows10®, Linux®, Unix®-based operating systems, or other OS. Missionprocessing can be offloaded from the flight controller 114 to thecomputing device 112. In various embodiments, the scanning manager 115can execute on the computing device 112, client device 110, payload 124,a remote server (not shown), or other computing device. In someembodiments, computing device 112 and client device 110 may be the samedevice.

As discussed, image data collected by scanning sensor 130 on the groundvehicle can be transformed into a birds-eye view using perspectivetransform techniques, such as using a homography. This leads to adistorted birds-eye view, which may lose details of various roadwayobjects, e.g., the parallelism of lane markings, blurred markings, etc.,which may lead to errors in identifying the lane in which the groundvehicle 111 is traveling. Instead of relying on transforming afront-view image to a birds-eye view image, embodiments can utilizebirds-eye view scanning data obtained from movable object 104. In someembodiments, scanning manager 126 can be used to collect scanning datafrom the roadway environment in which the ground vehicle 111 isoperating. In some embodiments, the scanning data can include image dataof the roadway in front of, or around, the ground vehicle 111. Thescanning data may be collected at or near a location instructed by theclient device 110 (e.g., satellite navigation system coordinates, aposition relative to the ground vehicle, etc.). The image data can betransmitted by the scanning manager 126 to the detection manager 128 ofthe ground vehicle 111.

In some embodiments, the detection manager can obtain scanning data fromscanning sensor 130. The scanning data from scanning sensor 130 mayinclude front-view image data from front-mounted scanning sensor 130. Insome embodiments, scanning sensor 130 may be mounted to ground vehicle111 to capture rear-view image data, side-view image data, up-view imagedata, or image data in any suitable direction. The detection manager 128can identify an overlapping portion of the scanning data received fromthe movable object and the ground vehicle using reference objects in thescanning data. The reference objects may include road signs, lightpoles, trees, or other identifiable objects in the roadway environment.In some embodiments, the reference object may be a coordinates in theroadway environment (e.g., GPS coordinates), which correspond to a pointin the scanning data corresponding. These coordinates may representcorrespond to any point viewable by both the ground vehicle and theaerial vehicle such as a point directly below movable object 104.Because the scanning data captured by the movable object does notinclude any transformation-related distortions, the scanning data may beused to perform lane detection to a longer distance in front of theground vehicle 111, compared to the transformed forward perspective(e.g., front-view) scanning data. In some embodiments, the front-viewimage data can be transformed into an overhead view (e.g., birds-eyeview) and an overlapping portion between the scanning data captured bythe movable object and the transformed birds-eye view can be identified.The scanning data captured by the movable object can then be used tocalibrate the transformed birds-eye view based on the overlappingportion. In various embodiments, scan matching techniques, featurematching, or other alignment techniques may be used to align thescanning data and identify the overlapping portion.

FIGS. 2A-2C illustrates an example of scanning data of a roadwayenvironment obtained from an aerial vehicle and a ground vehicle, inaccordance with various embodiments. As shown in FIG. 2A, the roadwayenvironment 200 can include the ground vehicle driving on a roadway andan aerial vehicle flying above the roadway. As discussed, each vehiclecan include one or more scanning sensors to collect scanning data fromthe roadway environment. One use of the scanning data is to performvarious autonomous driving and/or assisted driving functions, such aslane detection.

As discussed, the scanning data captured by a scanning sensor on groundvehicle 111 can be captured from the vehicle-view angle, as shown inFIG. 2B at 206. As shown in FIG. 2B, the vehicle-view perspectiveresults in the lanes converging. The vehicle view can be transformedinto a birds-eye view using perspective transform techniques. However,this can lead to distortions of the scanning data. For example,depending on the perspective transform used, the portions of the roadwayenvironment 200 farther from the scanning sensor of the ground vehiclecan become blurry and may no longer appear parallel in the transformedscanning data. Depending on the lane detection technique used, thesedistortions may cause the lane to be misidentified. Likewise, otherroadway objects, such as other vehicles, signs, environmental obstacleslike trees, and other such objects may also be misidentified in thetransformed image data.

However, in accordance with an embodiment, as shown at 208, a birds-eyeview of the roadway environment 200 can be obtained directly from aerialvehicle 104. The scanning data obtained from the aerial vehicle does notrequire transformation to show the roadway environment from a birds-eyeview. As such, the scanning data does not have the distortions caused byperspective transformation.

Accordingly, in various embodiments, the ground vehicle 111 can use thescanning data (e.g., image data, point cloud data, etc.) captured by theaerial vehicle 104 to perform autonomous driving and/or assisted drivingfunctions. For example, the ground vehicle can perform lane detectionusing the scanning data received from the aerial vehicle. Lane detectionmay be performed as is generally known, such as through using, e.g.,Canny edge detection to identify lane markings in the scanning data thatindicate lane boundaries. Additionally, or alternatively, binary imageanalysis may be performed. A binary image can be obtained by utilizinggradient and color threshold values. Each pixel in the scanning data canbe analyzed and, when the value of pixel is higher than the thresholdvalue, that pixel can be set as 1 and when the pixel is lower than thethreshold value, that pixel can be set to zero. A sliding window may berun on the resulting binary image, to calculate a polynomial curve tofit the lane represented in the binary image. Because the scanning datareceived from the aerial vehicle has not been transformed, and istherefore not distorted due to a perspective transformation, the lanedetection performed on this scanning data provides more accurate lanedetection results.

The scanning data received from the ground vehicle can be calibrated tomake use of the more accurate lane detection information obtained fromanalyzing the scanning data received from the aerial vehicle. Forexample, as shown in FIGS. 2A-2C, a reference object 202 observable toboth the aerial vehicle and ground vehicle can be chosen to performcalibration. The reference object 202 can be a physical object in theroadway environment, such as a tree (as shown), road sign, pole, orfixed roadway object. Additionally, or alternatively, the referenceobject 202 can be a point 204 in the roadway environment (e.g.,particular GPS coordinates). For example, as shown in FIG. 2A, thereference object can be a point 204 (e.g., a global position or aposition relative to the ground vehicle or aerial vehicle) which isdirectly below the aerial vehicle.

As shown in FIG. 2B, reference object 202 can be identified in thevehicle scanning data. The reference object can be associated with acoordinate in the scanning data (e.g., an image coordinate system basedon pixels). For example, the example front-view shown in FIG. 2B is aprojection of the roadway environment 200 onto a two-dimensional imageplane. Using the intrinsic parameters (e.g., focal length, lensparameters, etc.) and extrinsic parameters (e.g., position, orientation,etc.) of the scanning sensor of the ground vehicle, a projection matrixcan be used to convert from the image coordinate system of thentwo-dimensional image plane to the world coordinate system of thethree-dimensional roadway environment. Similarly, a perspectivetransformation can be used to convert from the front-view image plane toa birds-eye image plane. These transformations allow positions in thetransformed birds-eye view to be mapped to the front-view and to theroadway environment 200. For a physical reference object, the pixelsrepresenting the reference object in the scanning data can beidentified. In some embodiments, image processing techniques and/ormachine learning techniques, such as object recognition, may be used toidentify the reference object in the scanning data. The pixelsrepresenting the reference object in the scanning data can be mapped tothe transformed birds-eye view based on the scanning data.

As discussed, the reference object can be a position directly under theaerial vehicle, or other point, in the roadway environment. The aerialvehicle can determine its position using a positioning sensor, such as aglobal navigation satellite service (GNSS) sensor, e.g., GPS, BeiDou,etc. The position of the reference object can be the position of theaerial vehicle with the elevation value zeroed out (or set to anothervalue, such as the elevation value of the ground vehicle). In someembodiments, the aerial vehicle can provide its position to the groundvehicle. In some embodiments, the ground vehicle can identify theposition of the aerial vehicle relative to the position of the groundvehicle using one or more sensors coupled to the ground vehicle. Thesensors may include an infrared range finder, ultrasonic rangefinder,radar sensor, LiDAR sensor, etc. The ground vehicle can use the sensordata to determine the location of the aerial vehicle relative to theground vehicle. The ground vehicle can include a GNSS sensor used todetermine the current position of the ground vehicle. Using its currentposition and the relative position of the aerial vehicle, the groundvehicle can determine a position of the aerial vehicle in the worldcoordinate system of the roadway environment. As discussed, the groundvehicle can convert a position in the world coordinate system to theimage coordinate system of the front-view image 206. Once converted tothe image coordinate system, the ground vehicle can map the positionfrom the image coordinate system to the transformed birds-eye viewcoordinate system.

In some embodiments, when the aerial vehicle transmits its birds-eyeview 208, the ground vehicle can identify an overlapping portion of thebirds-eye view 208 captured by the aerial vehicle and the transformedbirds-eye view based on the front-view 206 captured by the groundvehicle. For example, the position of the reference object in bothimages can be used to align the two images and/or to map the aerialvehicle's scanning data to the ground vehicle's scanning data. In someembodiments, the aerial vehicle may provide scanning data that extendsfarther in front of the ground vehicle than the ground vehicle'sscanning data shows. The ground vehicle can therefore extend its view ofthe roadway environment using the scanning data from the aerial vehicle,aligned using the reference object.

In some embodiments, the ground vehicle 111 can instruct the aerialvehicle to capture scanning data at or near particular positions in theroadway environment, using either absolute coordinates (e.g., usingglobal navigation satellite service coordinates) or a position relativeto the ground vehicle. For example, the ground vehicle can send controldata to the aerial vehicle with a position 100 meters ahead of theground vehicle. The aerial vehicle can process the control data and theaerial vehicle can generate movement commands that are send to themovement mechanisms 116 of the aerial vehicle which causes the aerialvehicle to travel to the instructed position. The aerial vehicle canthen capture scanning data at the instructed position and transmit thescanning data to the ground vehicle for further processing.

FIG. 3 illustrates an example 300 of a scanning manager and a detectionmanager, in accordance with various embodiments. As shown in FIG. 3, ascanning manager 126 may execute on one or more processors 302 ofcomputing device 112. The one or more processors 302 may include CPUs,GPUs, FGPAs, SoCs, or other processors. The scanning manager 126 mayinclude sensor interfaces 303 and capture manager 308.

Sensor interfaces 303 can include a scanning sensor interface 304 and apositioning sensor interface 306. The sensor interfaces 303 may includehardware and/or software interfaces. The scanning sensor interface 304can receive data from the scanning sensor (e.g., an imaging device, aLiDAR sensor, or other scanning sensor) and the positioning sensorinterface 306 can receive data from a positioning sensor (e.g., a GNSSsensor, RTK sensor, etc.). In various embodiments, the scanning sensormay produce data in a point cloud format. The point cloud data may be athree-dimensional representation of the target environment. In someembodiments, the point cloud data may be converted to a matrixrepresentation. In some embodiments, the scanning sensor may produceimage data, such as high definition images and/or video of the roadwayenvironment in which the aerial vehicle is traveling. The positioningdata may include GPS coordinates for the aerial vehicle.

In some embodiments, control data can be received from client device 110instructing the aerial vehicle to move to a particular position. Thecontrol data can be received via communication system 120B and sentdirectly to flight controller 114 to be used to generate movementcommands to cause the aerial vehicle to move to the instructed position.In some embodiments, the control data, or at least the instructedposition, may be passed to the capture manager 308 by the communicationsystem 120B or flight controller 114. The capture manager 308 caninclude a position monitor 310 which monitors a current position of theaerial vehicle based on position information received throughpositioning sensor interface 306. Position monitor 310 can furthercompare the instructed position to the current position of the aerialvehicle. For example, position monitor may implement hardware orsoftware logic to compare two input values (e.g., current coordinates ofthe aerial vehicle and target coordinates of the aerial vehicle) andoutput a binary value indicating whether the two input values match. Ifthe two input values match, a signal can be sent to scanning trigger 312which may then send an instruction to the scanning sensor via scanningsensor interface 304 to cause the scanning sensor to capture scanningdata of the roadway environment. As discussed, this may include imagedata captured using an imaging device and/or point cloud data capturedusing a LiDAR sensor, and/or other scanning data of the roadwayenvironment. Once the scanning data has been captured, the scanning datacan be transmitted to the client device 110 for further processing.

Client device 110 may include a detection manager 128 executing on oneor more processors 314. The one or more processors 302 may include CPUs,GPUs, FGPAs, SoCs, or other processors. The detection manager 128 mayinclude sensor interfaces 316 and scanning data processor 317. Thesensor interfaces can include hardware or software interfaces thatfacilitate communication of data and/or commands between the detectionmanager 128 and various scanning sensors and positioning sensors coupledto ground vehicle 111. Scanning data processor 317 can include a datamanager 322 that receives scanning data from scanning manager 126 viacommunication system 120A, and from onboard scanning sensors coupled toground vehicle 111 via scanning sensor interface 318. The scanning datafrom scanning manager 126 may include positioning data representing aposition of the aerial vehicle when the scanning data was captured. Insome embodiments, the data manager 322 may further collect position datafrom the positioning sensors onboard the ground vehicle via positioningsensor interface 320.

Data manager 322 can associate the scanning data received from thescanning manager with the scanning data received from the groundvehicle. For example, pairs of scanning data may be timestamped to beassociated with one another by the data manager. The associated data maybe passed to an image transform manager 324. As discussed, imagetransform manager can implement perspective transformation techniques toconvert the front-view scanning data collected by the ground vehicle toa birds-eye view. The image transform manager 324 can identify anoverlapping portion of the scanning data captured by the aerial vehicleand the transformed birds-eye view based on the front-view scanning datacaptured by the ground vehicle. For example, the position of a referenceobject in both images can be used to align the two images and/or to mapthe aerial vehicle's scanning data to the ground vehicle's scanningdata, as discussed above. In some embodiments, matching can be performedbetween the scanning data captured by the aerial vehicle and thefront-view scanning data captured by the ground vehicle. If the datamatches, and/or if an overlap is identified, then data does not need tobe transformed before being matched with the overhead image. If they domatch, then the aerial scanning data can be used instead of transformingthe front-view scanning data. Alternatively, in some embodiments, thefront-view scanning data may be transformed to an overhead view, and theaerial scanning data may be used to calibrate the transformed overheadview (e.g., to correct for distortions).

Once the scanning data received from the aerial vehicle has been matchedwith the transformed scanning data of the ground vehicle, the scanningdata from the aerial vehicle can be passed to object detector 326.Object detector 326 may implement various object detection techniques,such as an object detection machine learning model trained to identifyroadway objects in the scanning data and/or edge detection techniquesused to identify roadway objects such as lane markings. When roadwayobjects are identified in the scanning data received from the aerialvehicle, the positions of the objects can be mapped to theircorresponding positions in the scanning data captured by the groundvehicle. In some embodiments, navigation commands can be generated basedon the object identified in the scanning data. For example, when a laneis detected in the scanning data, a polynomial curve can be calculatedthat fits the detected lane in the scanning data. The polynomial curvecan be used to determine a trajectory for the ground vehicle to followto stay within the lane. This can be provided to navigation controller328 which can generate movement commands for the ground vehicle.

In some embodiments, object detector 326 may determine that thecollected scanning data provides insufficient range ahead of the groundvehicle to identify objects in enough time to determine appropriatenavigation commands. For example, weather conditions may limitvisibility, or the roadway ahead may be winding, limiting the effectiverange of the scanning sensors of the ground vehicle. In someembodiments, the object detector may include a machine learning modeltrained to determine an optimal scanning range (e.g., distance in frontof the ground vehicle) at which to collect scanning data. The model maybe trained based on roadway conditions (weather, traffic, etc.) and/ordriving conditions (current speed, direction, etc.). In someembodiments, the model may be trained on past navigation data for thecurrent location of the ground vehicle collected by other groundvehicles that have traveled in the area. When the object detectordetermines that a different scanning range is needed, the objectdetector can send a request to the navigation controller 328 to instructthe aerial vehicle to move to a new position to capture scanning data atthe new scanning range. The navigation controller 328 can send controldata to the aerial vehicle via communication system 120A, and inresponse the aerial vehicle may reposition itself to the instructedposition.

Although embodiments are discussed with respect to an aerialvehicle-ground vehicle pair, in various embodiments, one aerial vehiclemay provide scanning data to a plurality of ground vehicles.Additionally, in some embodiments, a plurality of aerial vehicles may bein communication with the ground vehicle, with each aerial vehiclecapturing scanning data at a different position and transmitting thescanning data to the ground vehicle. The ground vehicle can use thescanning data to construct a local map around the ground vehicle (e.g.,using Simultaneous Localization and Mapping (SLAM) techniques). In someembodiments, the aerial vehicle may be stored in the ground vehicle. Forexample, the ground vehicle may include a platform from which the aerialvehicle may launch from and land on. The ground vehicle may, forexample, determine that current driving conditions require additionalscanning data and can launch the aerial vehicle to collect it. Whenconditions improve, or when the aerial vehicle requires additionalcharging, the aerial vehicle can return to the ground vehicle forcharging and storage.

FIG. 4 illustrates an example of a machine learning model for use inroadway object detection, in accordance with various embodiments. Asdiscussed, in various embodiments, scanning data can be obtained by theaerial vehicle and the ground vehicle. The front-view scanning dataobtained by the ground vehicle can be transformed into an intermediateformat (e.g., a birds-eye view) which enables the scanning data capturedby the ground vehicle to be calibrated using the birds-eye view scanningdata captured by the aerial vehicle. Calibration may include bothaligning the intermediate format with the scanning data captured by theaerial vehicle (e.g., based on the position of one or more referenceobjects in both sets of data) as well as augmentation of theintermediate format (e.g., adding corresponding depth informationincluded in the scanning data obtained by the aerial vehicle to theoverlapping portion of the intermediate format). In various embodiments,the intermediate format can be provided as input to a machine learningmodel to perform object detection, pattern recognition, or otherinference on the scanning data.

In some embodiments, each image obtained by the aerial vehicle ismatched with the intermediate format at the pixel level (e.g., where atleast a portion of the scanning data obtained by the aerial vehicle andthe ground vehicle overlaps). A machine learning model, such asconvolutional neural network (CNN) 400, can be trained to identifysemantic information associated with each pixel, or patch of pixels, inthe intermediate format. For example, a training dataset may bemaintained that includes ground truth labels (e.g., lane marking, road,vehicle, traffic barrier, etc.). The CNN 400 can be trained on thetraining dataset until the output of the CNN is within a thresholdconfidence of the ground truth labels. In some embodiments, the input tothe CNN can be represented as N*4*H*W, where N corresponds to the numberof input patches in the input image data, 4 corresponds to the number ofchannels in the input image data (e.g., each pixel may includeR+G+B+depth values), H corresponds to the height of the input imagedata, and W corresponds to the width of the input image data. In someembodiments, depth can correspond to the depth of field measured by theaerial vehicle which has been added to the intermediate format. In someembodiments, CNN 400 may include a plurality of layers 406-412, witheach layer including a convolution layer (CONV), a batch normalizationlayer (BN), and an activation function layer (RELU). In someembodiments, the CNN 400 may implement a skip connection architecture,in which some data output by one layer can be provided to the next layerand can be provided to a further layer, as depicted at 414. This allowssome data that might otherwise be abstracted away to be retained for useby other layers in the CNN. Although one particular architecture isdepicted in FIG. 4, various machine learning models may be used inaccordance with an embodiment.

When performing inference, the input image data 402 (e.g., thetransformed and calibrated intermediate format representing thebirds-eye view of the roadway environment) can be provided to thetrained CNN 400. The CNN 400 can analyze the input image data 402 andprovide an output 404 which may be represented as N*K*H*W, wherein forthe N*W pixels of the input N patches, the semantic information of Kchannels is output. The output value of each channel represents theprobability of the semantic information of the channel. For example, fora certain pixel, the output may be lane marking 80%, road 13%, vehicle1%. As a result, the content of each pixel in the input image data canbe characterized. This allows, e.g., each pixel of the input imageshowing a lane marking to be identified. A curve can then be fit tothose pixels to identify the current lane in which the vehicle istraveling. The ground vehicle can then reverse the projection from theintermediate format to the front-view and map those pixels that includelane markings to the front view. The resulting curve can be used togenerate movement commands that cause the ground vehicle to follow atrajectory that keeps the ground vehicle in the lane.

As discussed above, in some embodiments, a reference object depicted inthe scanning data of both the aerial vehicle and the ground vehicle canbe used to calibrate the scanning data of the ground vehicle. To accountfor differences in lighting between the two perspectives, varioustechniques, such as Zero Mean Normalized Cross-Correlation (ZNCC),semi-global matching (SGM), and sweeping may be used to compensate forradiometric variations between the input scanning data. In someembodiments, k-means clustering, or other clustering techniques can beused to quickly identify features in the scanning data. The features canbe assigned a weight based on likely importance. In some embodiments,calibration can be performed using the position of the reference objectin the scanning data. The pixel or pixels corresponding to the referenceobject can be assigned a label indicating they are associated with thereference object. This provides pixel level calibration to the scanningdata.

As discussed, in some embodiments, the scanning data may include pointcloud data. The point cloud data may be a three-dimensionalrepresentation of the target environment (e.g., the roadwayenvironment). This 3D representation can be divided into voxels (e.g.,3D pixels). Each point in the point cloud of the mapping data isassociated with a position in the scanner reference frame that isdetermined relative to the scanning sensor. The positioning data of themovable object, produced by the positioning sensor, may then be used toconvert this position in the scanner reference frame to the outputreference frame in a world coordinate system. In some embodiments, thescanning data may be encoded before it is transmitted from the aerialvehicle to the ground vehicle. For example, the scanning data obtainedby the scanning sensor on the aerial vehicle may be a denserepresentation, which may be transformed into a sparse representationprior to transmitting to the ground vehicle. In some embodiments, thedifference between each frame of the scanning data may be transmitted,rather than the entire frame of scanning data. In some embodiments, thepoint cloud data obtained by the aerial vehicle can be used to augmentthe point cloud data obtained by the ground vehicle. For example, the Xand Y components of the point-cloud for objects that are far away fromthe ground vehicle can be captured more accurately by the aerialvehicle. This data can be shared with the ground vehicle, improving theaccuracy of the calculation of the ground vehicle mounted threedimensional point-cloud.

In some embodiments, the aerial vehicle and the ground vehicle may eachbe configured to map the roadway environment using SimultaneousLocalization and Mapping (SLAM) techniques to generate a local map. TheSLAM techniques may use data captured by the scanning sensors and/orvarious other sensors, such as as an IMU, a gyroscope, or other suitablesensors. The SLAM generated by the aerial vehicle can be transmitted tothe ground vehicle and combined with the SLAM generated by the groundvehicle to extend the local map generated by the ground vehicle. In someembodiments, traffic conditions (e.g., weather conditions, trafficdensity, etc.) may limit the ability of the ground vehicle to generate amap using SLAM techniques. When such traffic conditions are detected,the ground vehicle can instruct the aerial vehicle to capture birds-eyeview visual data to complete the local map generated by the groundvehicle.

FIG. 5 shows a flowchart of a method of sharing sensor informationbetween multiple vehicles in a movable object environment, in accordancewith various embodiments. At 502, first scanning data can be receivedfrom a first computing device coupled to an aerial vehicle, by a secondcomputing device included in a ground vehicle. The first scanning datais obtained using a first scanning sensor coupled to the aerial vehicle.In some embodiments, the first scanning data includes first mapping datagenerated based on point cloud data collected by the first scanningsensor.

At 504, obtaining second scanning data from a second scanning sensorcoupled to the ground vehicle. In some embodiments, the first scanningdata can be received by a plurality of ground vehicles in communicationwith the aerial vehicle. In some embodiments, scanning data can beobtained by a plurality of aerial vehicles in communication with theground vehicle, and the scanning data can be transmitted by theplurality of aerial vehicles to the ground vehicle. In some embodiments,the first scanning sensor includes a first LiDAR sensor and the secondscanning sensor includes a second LiDAR sensor. In some embodiments, thesecond scanning data includes second mapping data generated based onpoint cloud data collected by the second scanning sensor. In someembodiments, the first mapping data can be combined with the secondmapping data to increase a coverage area of a local map maintained bythe ground vehicle.

At 506, an overlapping portion of the first scanning data and the secondscanning data can be identified based on at least one reference objectin the first scanning data and the second scanning data. In someembodiments, the at least one reference object includes a representationof a roadway object represented in the first scanning data and thesecond scanning data.

At 508, a navigation control command can be executed based on one ormore navigation objects identified in the overlapping portion of thefirst scanning data and the second scanning data. In some embodiments,the one or more navigation objects include a driving lane represented inthe overlapping portion of the first scanning data and the secondscanning data.

In some embodiments, the method may further include receiving a movementcommand from the second computing device on the ground vehicle, themovement command including a location, causing the aerial vehicle tomove to the location, and obtaining the first scanning data at or nearthe location. In some embodiments, the location includes at least one oflocation coordinates, a position relative to the ground vehicle or otherobject.

In some embodiments, the method may further include, transforming thefirst scanning data and the second scanning data into an intermediateformat, performing object recognition on the intermediate format using amachine learning model to identify the one or more navigation objects,the machine learning model trained to identify the one or morenavigation objects, and determining the navigation control command basedon the object recognition. In some embodiments, an input to the machinelearning model includes a plurality of input patches from theintermediate format, a number of channels, and a height and width of theinput patches, the number of channels representing color and depthassociated with each input patch. An output of the machine learningmodel includes one or more confidence scores for each input patch, theone or more confidence scores associated with the one or more navigationobjects.

In some embodiments, the method may further include detecting a trafficcondition affecting the ground vehicle, and in response to the detectedtraffic condition, sending a command to the first computing device onthe aerial vehicle to collect the first mapping data.

Unmanned aerial vehicles (UAVs) carry computing devices and sensorswhich enable UAVs to implement Simultaneous Localization and Mapping(SLAM) to map their environment. A UAV can scan its environment usingscanning sensors such as imaging sensors, LiDAR sensors, etc., whileflying and construct a localization mapping for itself to plan a route.The resulting map may also be used for obstacle avoidance andauto-piloting. However, UAVs are still limited in the quantity andsophistication of the computing devices, sensors, batteries, etc., thatthey can carry, which results in reduced capability to use SLAMtechniques to generate large, accurate maps. Additionally, whilecollecting data to perform SLAM, the aerial vehicle is subject to wind,user-initiated movements, and other instabilities which can limit theaccuracy of the data collected. Additionally, some UAVs may move tooquickly for an accurate SLAM to be used for navigation.

SLAM techniques provide a way of construction and/or updating areal-time map of the environment in which a movable object is operatingand simultaneously maintaining a position of the moveable object withinthe real-time map. deals with a computational problem of constructing orupdating a map of an unfamiliar environment while simultaneously keepingtrack of an agent's local with it. A typical SLAM deals withconstructing stereoscopic frames and connecting the frames to form acontinuous map. SLAM may be performed using imaging sensors, such asmonocular or binocular cameras, LiDAR sensors, or other scanningsensors.

Drones and other unmanned aerial vehicles (UAV) are limited in quantityand quality of sensors and computers they may carry. This can be due tothe costs of such equipment, weight restrictions, power restrictions,etc. Cars and other ground vehicles, however, are not nearly asrestricted in their capacity to carry sophisticated computers andsensors). A car provides a more protected and stable platform to carryexpensive equipment, and any additional weight or power requirements aregenerally easily accommodated. As a car drives along a roadway, it canuse its onboard sensors to generate a high precision local map. The carmay use cameras, LiDAR, inertial measurement units, gyroscopes, andother sensors to generate the high precision map. The car can transmitthe resulting high precision map to a drone flying near the car. Thedrone can also generate a map based on its onboard sensors as it fliesnearby. However, this is a lower precision map, due to the types ofsensors carried by the drone, and the computing resources available tothe drone. However, the drone can combine its map with the highprecision map received from the car to obtain a high precision map itcan use to navigate through its environment (for example to avoidobject, generate routes, etc. Embodiments take advantage of the higherquality sensors and computers available in a car or other ground vehicleto improve the maps available to a drone or other UAV.

Embodiments provide a map construction method that can use datacollected by a ground vehicle, as well as computing devices and sensorsavailable to the ground vehicle, to generate a more comprehensive andaccurate map than the UAV is capable of generating on its own. The UAVcan share data it collects an as well as any maps constructed using SLAMtechniques with the ground vehicle. The ground vehicle can combine itsown map with that of the UAV to generate a combined map that may beshared with the UAV. The combined map may include more details and covera larger area than the map generated by the UAV alone. The combined mapmay then be transmitted to the UAV for use in navigation, tailing andobstacle avoidance.

FIG. 6 illustrates an example 600 of an aerial vehicle and a groundvehicle, in accordance with various embodiments. As discussed above, aground vehicle 111 may include one or more scanning sensors 132, such asa LiDAR sensor. The ground vehicle's scanning sensors may have a sensingrange 602, 604 in which it can detect its environment. An aerialvehicle, 104, may also include one or more scanning sensors 124, withwhich it can scan its environment. As discussed above, the aerialvehicle 104 and ground vehicle 111 may both include various sensors,computing devices, movement mechanisms, and controllers which enable theaerial vehicle and ground vehicle to navigate their environment.

Using its scanning sensors 132, the ground vehicle can obtain a positionof the aerial vehicle so long as it is operating within range of theground vehicle's sensors. The position may be relative to the groundvehicle or may be an absolute position in the world coordinate system orother coordinate system. Both the aerial vehicle 104 and the groundvehicle 111 can use their respective sensors to perform SLAM of theenvironment in which they are operating. As discussed further, theground vehicle can perform SLAM using improved sensors and computingdevices when compared to those available to the aerial vehicle. Theground vehicle can therefore generate a more accurate SLAM compared tothe aerial vehicle and provide the map to the aerial vehicle for use innavigation. In some embodiments, the aerial vehicle may also map theenvironment using SLAM techniques, which may result in a lower precisionmap compared to that produced by the ground vehicle. The aerial vehiclecan transmit its map to the ground vehicle, which may then combine theaerial vehicle's map with that generated by the ground vehicle, andreturn the higher precision combined map to the aerial vehicle for usein navigation.

FIG. 7 illustrates an example 700 of generating a map of a movableobject environment using an aerial vehicle and a ground vehicle, inaccordance with various embodiments. As shown in FIG. 7, At 702, groundvehicle 111 can carry out SLAM with ground vehicle scanning sensor 132.In various embodiments, SLAM techniques may be used to generate a localmap using data from various sensors, including the scanning sensors andother sensors such as an IMU, a gyroscope, and other suitable sensors.The scanning sensor may be an imaging sensor or a LiDAR sensor, whichcan be used to obtain a real-time map of the environment in the viewangle of the ground vehicle 111. With the advantage of the vehiclemounted sensors and processors, this real-time construction map is of ahigher precision than a real-time map constructed by the aerial vehicle104. In addition to the improved sensors and processing capabilities ofthe ground vehicle, it also provides a more stable platform from whichto collect the visual or LiDAR data of the environment, providing a moreaccurate real-time map.

In some embodiments, the ground vehicle can request the position of theaerial vehicle from the aerial vehicle. The aerial vehicle may provideits position in GNSS coordinates or a real-time relative positionobtained through combining the initial position and the real-timemovement state of the aerial vehicle. In some embodiments, the groundvehicle may obtain a relative position of the aerial vehicle directlyusing a scanning sensor 132 or other scanning sensor coupled to theground vehicle, such as a radar sensor, time of flight (ToF) camera, orother sensor. Using the position of the aerial vehicle, the groundvehicle can construct the real-time map and transmit it to the aerialvehicle at 704. In some embodiments, the ground vehicle may also provideits position to the aerial vehicle along with the real-time map.

At 706, the aerial vehicle can perform SLAM and positioning using itsonboard scanning and positioning sensors. However, the resultingreal-time map generated by the aerial vehicle may be of relatively lowprecision compared to the real-time map obtained from the groundvehicle. In some embodiments, the aerial vehicle can identify anoverlapping portion of real-time maps and combine the real-time mapsinto a combined real-time map.

In some embodiments, the aerial vehicle may transmit its real-time mapto the ground vehicle. The ground vehicle can then identify theoverlapping portion of the real-time maps and combine them into acombined real-time map. In various embodiments, the real-time maps maybe merged using scan matching techniques, or through matching featuresidentified in each real-time map to identify the overlapping portion ofthe maps. The real-time maps may then be merged by mapping locations ineach map to the other based on the overlapping portion. In someembodiments, the aerial vehicle may construct a real-time map using animaging sensor, while the ground vehicle may construct a real-time mapusing a LiDAR sensor (or vice versa). These real-time maps may be mergedby the ground vehicle, by identifying features in the image-basedreal-time map to features in the point cloud-based real-time map. Forexample, the aerial vehicle can transmit visual image informationobtained by its imaging sensors to the ground vehicle. The groundvehicle can convert the coordinate system of the image-based real-timemap to match the coordinate system of the point cloud data collected bythe ground vehicle. The ground vehicle can then return a combined map ofthe matched overlapping portion to the aerial vehicle to be used fornavigation.

FIG. 8 illustrates an alternative example 800 of generating a map of amovable object environment using an aerial vehicle and a ground vehicle,in accordance with various embodiments. As shown in FIG. 8, the groundvehicle 111 can generate a high precision SLAM using its onboardscanning sensor(s) 132, as shown at 802. The aerial vehicle 104 cancapture real-time image data and transmit it to the ground vehicle at804. At 806, The ground vehicle can then generate a map using both thevisual data obtained from the aerial vehicle and the SLAM generated bythe ground vehicle. This offloads all map generation from the aerialvehicle to the ground vehicle, with the aerial vehicle only providedreal-time image data. In some embodiments, the ground vehicle cancontrol the aerial vehicle using the combined map it generated, byproviding control data to the aerial vehicle to perform objectavoidance, tailing, and other navigation tasks.

In some embodiments, the aerial vehicle 104 can provide overheadscanning data to the ground vehicle 111 (e.g., as described above withrespect to FIGS. 1-5). The overhead scanning data can be used by theground vehicle in addition, or as an alternative, to the scanning datacaptured by the ground vehicle's onboard sensors to generate the SLAM.The ground vehicle can then provide the resulting SLAM to the aerialvehicle 111 and/or provide control data to the aerial vehicle based onthe resulting SLAM to perform various navigation tasks.

FIG. 9 illustrates an example of collaborative map construction by anaerial vehicle mapping manager and a ground vehicle mapping manager, inaccordance with various embodiments. As shown in FIG. 9, an aerialmapping manager 900 may execute on one or more processors 302 ofcomputing device 112. The one or more processors 302 may include CPUs,GPUs, FGPAs, SoCs, or other processors. The aerial mapping manager 900may include sensor interfaces 303 and map generator 904.

Sensor interfaces 303 can include a scanning sensor interface 304 and apositioning sensor interface 306. The sensor interfaces 303 may includehardware and/or software interfaces. The scanning sensor interface 304can receive data from the scanning sensor (e.g., an imaging device, aLiDAR sensor, or other scanning sensor) and the positioning sensorinterface 306 can receive data from a positioning sensor (e.g., a GNSSsensor, RTK sensor, etc.). In various embodiments, the scanning sensormay produce data in a point cloud format. The point cloud data may be athree-dimensional representation of the target environment. In someembodiments, the point cloud data may be converted to a matrixrepresentation. In some embodiments, the scanning sensor may produceimage data, such as high definition images and/or video of the roadwayenvironment in which the aerial vehicle is traveling. The positioningdata may include GPS coordinates for the aerial vehicle.

In some embodiments, map generator 904 can include a data manager 908which can obtain scanning data via the scanning sensor interface fromthe scanning sensors onboard the aerial mapping manager. The datamanager can also obtain real-time position data from the aerialvehicle's positioning sensor via positioning sensor interface 306. Thedata manager 908 can pass the scanning data and position data to a SLAMmodule that implements one or more SLAM techniques. The SLAM module 910can output a real-time map of the environment in which the aerialvehicle is traveling as well as the aerial vehicle's position withinthat map. As discussed, the real-time map generated by SLAM module 910may be a relatively less precise map due, e.g., to the performancecapabilities of the onboard sensors, the computing device 112, stabilityof the aerial vehicle in flight, etc. Once the real-time map has beengenerated, the map generator can pass the real-time map to the groundmapping manager on client device 110 via communication system 120B. Asdiscussed, in some embodiments, the map generator 904 may output onlyscanning data, without first generating a real-time map using SLAMmodule 910. For example, resource manager 912 can monitor computingresource availability on computing device 112. If the availableresources (e.g., memory, storage capacity, battery level, processorutilization, etc.) is below a threshold value, the resource manager 912can instruct the data manager to provide the scanning data and positiondata to the ground mapping manager in a request to perform SLAM onbehalf of the aerial vehicle.

Client device 110 may include a ground mapping manager 902 executing onone or more processors 314. The one or more processors 302 may includeCPUs, GPUs, FGPAs, SoCs, or other processors. The ground mapping manager902 may include sensor interfaces 316 and map generator 906. The sensorinterfaces can include hardware or software interfaces that facilitatecommunication of data and/or commands between the ground mapping manager902 and various scanning sensors and positioning sensors coupled toground vehicle 111. Map generator 906 can include a data manager 914that receives scanning data from aerial mapping manager 900 viacommunication system 120A, and from onboard scanning sensors coupled toground vehicle 111 via scanning sensor interface 318. The data manager914 may also include position data for the ground vehicle viapositioning sensor interface 306. In some embodiments, the data manager914 may also receive position data for the aerial vehicle via scanningsensor interface 304. For example, ground vehicle 111 may determine therelative position of the aerial vehicle from the ground vehicle using anonboard LiDAR sensor, or other scanning sensor. The scanning data fromscanning manager 126 may include positioning data representing aposition of the aerial vehicle when the scanning data was captured.

The data manager 914 can pass the scanning data and the position data toSLAM module 916 to generate a real-time map of the environment in whichthe ground vehicle is operating. SLAM module 916 may implement variousSLAM techniques to generate the real-time map. Due to the relativelyhigher quality scanning sensors, the stability of the ground vehicleplatform, and available computing resources, the SLAM module 916 canoutput a relatively higher precision real-time map. In some embodiments,the real-time map generated by SLAM module 916 and the real-time mapgenerated by SLAM module 910 can be provided to map merge module 918.Map merge module 918 can implement scan matching techniques, featurematching techniques, or other image processing techniques to merge thetwo real-time maps. For example, an overlapping portion of the tworeal-time maps can be identified based on common features identified inboth maps. The resulting combined map can be provided to the aerialvehicle via communication system 120A to be used to perform variousnavigation operations. In some embodiments, only the overlapping portionof the two real-time maps is provided to the aerial vehicle. In someembodiments, SLAM module 916 can generate the real-time map based onscanning data obtained from the aerial vehicle directly, rather thanusing a SLAM map generated by the aerial mapping manager 900, andscanning data obtained from the ground vehicle. In such embodiments, theoutput real-time map can be provided to the aerial vehicle without theadditional merging step. In some embodiments, the combined real-time mapcan be used by the ground vehicle to control the aerial vehicle, withoutproviding the combined real-time map to the aerial vehicle. Instead, theground vehicle can generate navigation commands that are transmitted tothe aerial vehicle and converted by the aerial vehicle's flightcontroller into movement commands.

Although embodiments are discussed with respect to an aerialvehicle-ground vehicle pair, in various embodiments, one aerial vehiclemay provide scanning data to a plurality of ground vehicles.Additionally, in some embodiments, a plurality of aerial vehicles may bein communication with the ground vehicle, with each aerial vehiclecapturing scanning data at a different position and transmitting thescanning data to the ground vehicle. In some embodiments, the groundvehicle can be an autonomous vehicle. Alternatively, the ground vehiclecan be a manually driven vehicle that includes appropriate scanningsensors and position sensors as described herein.

FIG. 10 shows a flowchart of a method 1000 of collaborative mapconstruction in a movable object environment, in accordance with variousembodiments. At 1002, an aerial vehicle can receive a first real-timemap from a ground vehicle. The first real-time map is based on firstscanning data collected using a first scanning sensor coupled to theground vehicle. In some embodiments, receiving, by an aerial vehicle, afirst real-time map from a ground vehicle, wherein the first real-timemap is based on first scanning data collected using a first scanningsensor coupled to the ground vehicle, can include transmitting thesecond real-time map to the ground vehicle, the ground vehicleconfigured to convert coordinates in the second real-time map to acoordinate system to match the first real-time map, determine anoverlapping portion of the first real-time map and the second real-timemap in the coordinate system, and transmit with the overlapping portionto the aerial vehicle. In some embodiments, receiving the real-time mapcan include receiving position information of the aerial vehicle fromthe ground vehicle. In some embodiments, the position informationincludes a real-time relative position determined using the firstscanning sensor. In some embodiments, the position information includesglobal navigation satellite system (GNSS) received from the aerialvehicle. In some embodiments, the ground vehicle is an autonomousvehicle.

At 1004, a second real-time map can be obtained, the second real-timemap based on second scanning data collected using a second scanningsensor coupled to the aerial vehicle. In some embodiments, the firstreal-time map is a higher precision map than the second real-time map.At 1006, a third real-time map can be generated based on the firstreal-time map and the second real-time map. In some embodiments,generating a third real-time map based on the first real-time map andthe second real-time map, can include determining an overlapping portionof the first real-time map and the second real-time map, and merging thefirst real-time map and the second real-time map using the overlappingportion.

In some embodiments, the first scanning sensor includes a LiDAR sensorand the second scanning sensor includes a visual sensor. The firstreal-time map is constructed based on point cloud data obtained from thefirst scanning sensor, and the second real-time map is constructed basedon visual data obtained from the second scanning sensor.

In some embodiments, the method may further include determiningavailable resources associated with the aerial vehicle are below athreshold value, sending a request to the ground vehicle to generate thethird real-time map, the request including the second real-time map, andreceiving the third real-time map from the ground vehicle.

In some embodiments, the method may further include obtaining aplurality of real-time maps from a plurality of aerial vehiclescommunication with the ground vehicle, generating a fourth real-time mapbased on the first real-time map and each corresponding real-time mapfrom the plurality of aerial vehicles, and transmitting the fourthreal-time map to each of the plurality of aerial vehicles.

In some embodiments, the ground vehicle is configured to obtain thirdscanning data from a third scanning sensor coupled to the groundvehicle, the first scanning sensor including a LiDAR sensor and thethird scanning sensor including an imaging sensor, and generate thefirst real-time map based on the first scanning data and the thirdscanning data using a Simultaneous Localization and Mapping (SLAM)algorithm.

In some embodiments, the method may further include transmitting thesecond scanning data to the ground vehicle, wherein the ground vehicleis configured to generate the first real-time map based on the firstscanning data and the second scanning data using a SimultaneousLocalization and Mapping (SLAM) algorithm, wherein the first scanningsensor includes a LiDAR sensor and the second scanning sensor includesan imaging sensor.

In some embodiments, a system for generating a map based on sensorinformation from multiple vehicles can include an aerial vehicleincluding a first computing device and a first scanning sensor, and aground vehicle including a second computing device and a second scanningsensor. The first computing device may include at least one processorand a scanning manager, the scanning manager including firstinstructions which, when executed by the processor, cause the scanningmanager to obtain first scanning data from the first scanning sensorfrom an overhead perspective, and transmit the first scanning data tothe second computing device. The second computing device may include atleast one processor and a detection manager, the detection managerincluding second instructions which, when executed by the processor,cause the detection manager to receive the first scanning data from thefirst computing device, generate a local map based at least in part onthe first scanning data, and execute a navigation control command basedat least in part on the local map. As discussed, various SLAM techniquesmay be used to generate the local map based on data from multiplesensors including the scanning sensors and other sensors such as an IMU,gyroscope, etc.

In some embodiments, the second computing device can be furtherconfigured to obtain second scanning data from the second scanningsensor. The second scanning data is obtained from the second scanningsensor from a forward perspective. In some embodiments, to generate alocal map based at least in part on the first scanning data, the secondinstructions, when executed, further cause the detection manager totransform the second scanning data from the forward perspective to theoverhead perspective to obtain transformed second scanning data based atleast in part on the first scanning data, and generate the local mapbased on the transformed second scanning data. In some embodiments, togenerate a local map based at least in part on the first scanning data,the second instructions, when executed, further cause the detectionmanager to identify a portion of the first scanning data thatcorresponds to a portion of the second scanning data and generate thelocal map based on the identified portion of the first scanning data. Invarious embodiments, scan matching and other techniques as discussedabove may be used to identify the portion of the first scanning datathat corresponds to a portion of the second scanning data. The portionof the first scanning data and the portion of the second scanning datamay include a representation of a same area of a roadway environment.

FIG. 11 illustrates an example of supporting a movable object interfacein a software development environment, in accordance with variousembodiments. As shown in FIG. 11, a movable object interface 1103 can beused for providing access to a movable object 1101 in a softwaredevelopment environment 1100, such as a software development kit (SDK)environment. As discussed above, the scanning manager can be implementedusing an SDK or mobile SDK to enable applications to perform real-timemapping, as described herein.

Furthermore, the movable object 1101 can include various functionalmodules A-C 1111-1113, and the movable object interface 1103 can includedifferent interfacing components A-C 1131-1133. Each said interfacingcomponent A-C 1131-1133 in the movable object interface 1103 canrepresent a module A-C 1111-1113 in the movable object 1101.

In accordance with various embodiments, the movable object interface1103 can provide one or more callback functions for supporting adistributed computing model between the application and movable object1101.

The callback functions can be used by an application for confirmingwhether the movable object 1101 has received the commands. Also, thecallback functions can be used by an application for receiving theexecution results. Thus, the application and the movable object 1101 caninteract even though they are separated in space and in logic.

As shown in FIG. 11, the interfacing components A-C 1131-1133 can beassociated with the listeners A-C 1141-1143. A listener A-C 1141-1143can inform an interfacing component A-C 1131-1133 to use a correspondingcallback function to receive information from the related module(s).

Additionally, a data manager 1102, which prepares data 1120 for themovable object interface 1103, can decouple and package the relatedfunctionalities of the movable object 1101. Also, the data manager 1102can be used for managing the data exchange between the applications andthe movable object 1101. Thus, the application developer does not needto be involved in the complex data exchanging process.

For example, the SDK can provide a series of callback functions forcommunicating instance messages and for receiving the execution resultsfrom an unmanned aircraft. The SDK can configure the life cycle for thecallback functions in order to make sure that the informationinterchange is stable and completed. For example, the SDK can establishconnection between an unmanned aircraft and an application on a smartphone (e.g. using an Android system or an iOS system). Following thelife cycle of a smart phone system, the callback functions, such as theones receiving information from the unmanned aircraft, can takeadvantage of the patterns in the smart phone system and update thestatements accordingly to the different stages in the life cycle of thesmart phone system.

FIG. 12 illustrates an example of an unmanned aircraft interface, inaccordance with various embodiments. As shown in FIG. 12, an unmannedaircraft interface 1203 can represent an unmanned aircraft 1201. Thus,the applications, e.g. APPs 1204-1206, in the unmanned aircraftenvironment 1200 can access and control the unmanned aircraft 1201.

For example, the unmanned aircraft 1201 can include various modules,such as a camera 1211, a battery 1212, a gimbal 1213, and a flightcontroller 1214.

Correspondently, the movable object interface 1203 can include a cameracomponent 1221, a battery component 1222, a gimbal component 1223, and aflight controller component 1224.

Additionally, the movable object interface 1203 can include a groundstation component 1226, which is associated with the flight controllercomponent 1224. The ground station component operates to perform one ormore flight control operations, which may require a high-levelprivilege.

FIG. 13 illustrates an example of components for an unmanned aircraft ina software development kit (SDK), in accordance with variousembodiments. The SDK may correspond to all or a portion of the scanningmanager described above or may be used to implement the scanning manageras a standalone application. As shown in FIG. 13, the drone class 1301in the SDK 1300 is an aggregation of other components 1302-1307 for anunmanned aircraft (or a drone). The drone class 1301, which have accessto the other components 1302-1307, can exchange information with theother components 1302-1307 and controls the other components 1302-1307.

In accordance with various embodiments, an application may be accessibleto only one instance of the drone class 1301. Alternatively, multipleinstances of the drone class 1301 can present in an application.

In the SDK, an application can connect to the instance of the droneclass 1301 in order to upload the controlling commands to the unmannedaircraft. For example, the SDK may include a function for establishingthe connection to the unmanned aircraft. Also, the SDK can disconnectthe connection to the unmanned aircraft using an end connectionfunction. After connecting to the unmanned aircraft, the developer canhave access to the other classes (e.g. the camera class 1302 and thegimbal class 1304). Then, the drone class 1301 can be used for invokingthe specific functions, e.g. providing access data which can be used bythe flight controller to control the behavior, and/or limit themovement, of the unmanned aircraft.

In accordance with various embodiments, an application can use a batteryclass 1303 for controlling the power source of an unmanned aircraft.Also, the application can use the battery class 1303 for planning andtesting the schedule for various flight tasks.

As battery is one of the most restricted elements in an unmannedaircraft, the application may seriously consider the status of batterynot only for the safety of the unmanned aircraft but also for makingsure that the unmanned aircraft can finish the designated tasks. Forexample, the battery class 1303 can be configured such that if thebattery level is low, the unmanned aircraft can terminate the tasks andgo home outright.

Using the SDK, the application can obtain the current status andinformation of the battery by invoking a function to request informationfrom in the Drone Battery Class. In some embodiments, the SDK caninclude a function for controlling the frequency of such feedback.

In accordance with various embodiments, an application can use a cameraclass 1302 for defining various operations on the camera in a movableobject, such as an unmanned aircraft. For example, in SDK, the CameraClass includes functions for receiving media data in SD card, getting &setting photo parameters, taking photo and recording videos.

An application can use the camera class 1302 for modifying the settingof photos and records. For example, the SDK may include a function thatenables the developer to adjust the size of photos taken. Also, anapplication can use a media class for maintaining the photos andrecords.

In accordance with various embodiments, an application can use a gimbalclass 1304 for controlling the view of the unmanned aircraft. Forexample, the Gimbal Class can be used for configuring an actual view,e.g. setting a first personal view of the unmanned aircraft. Also, theGimbal Class can be used for automatically stabilizing the gimbal, inorder to be focused on one direction. Also, the application can use theGimbal Class to change the angle of view for detecting differentobjects.

In accordance with various embodiments, an application can use a flightcontroller class 1305 for providing various flight control informationand status about the unmanned aircraft. As discussed, the flightcontroller class can include functions for receiving and/or requestingaccess data to be used to control the movement of the unmanned aircraftacross various regions in an unmanned aircraft environment.

Using the Main Controller Class, an application can monitor the flightstatus, e.g. using instant messages. For example, the callback functionin the Main Controller Class can send back the instant message every onethousand milliseconds (1000 ms).

Furthermore, the Main Controller Class allows a user of the applicationto investigate the instance message received from the unmanned aircraft.For example, the pilots can analyze the data for each flight in order tofurther improve their flying skills.

In accordance with various embodiments, an application can use a groundstation class 1307 to perform a series of operations for controlling theunmanned aircraft.

For example, the SDK may require applications to have a SDK-LEVEL-2 keyfor using the Ground Station Class. The Ground Station Class can provideone-key-fly, on-key-go-home, manually controlling the drone by app (i.e.joystick mode), setting up a cruise and/or waypoints, and various othertask scheduling functionalities.

In accordance with various embodiments, an application can use acommunication component for establishing the network connection betweenthe application and the unmanned aircraft.

Many features of the present invention can be performed in, using, orwith the assistance of hardware, software, firmware, or combinationsthereof. Consequently, features of the present invention may beimplemented using a processing system (e.g., including one or moreprocessors). Exemplary processors can include, without limitation, oneor more general purpose microprocessors (for example, single ormulti-core processors), application-specific integrated circuits,application-specific instruction-set processors, graphics processingunits, physics processing units, digital signal processing units,coprocessors, network processing units, audio processing units,encryption processing units, and the like.

Features of the present invention can be implemented in, using, or withthe assistance of a computer program product which is a storage medium(media) or computer readable medium (media) having instructions storedthereon/in which can be used to program a processing system to performany of the features presented herein. The storage medium can include,but is not limited to, any type of disk including floppy disks, opticaldiscs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs,EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or opticalcards, nanosystems (including molecular memory ICs), or any type ofmedia or device suitable for storing instructions and/or data.

Stored on any one of the machine readable medium (media), features ofthe present invention can be incorporated in software and/or firmwarefor controlling the hardware of a processing system, and for enabling aprocessing system to interact with other mechanism utilizing the resultsof the present invention. Such software or firmware may include, but isnot limited to, application code, device drivers, operating systems andexecution environments/containers.

Features of the invention may also be implemented in hardware using, forexample, hardware components such as application specific integratedcircuits (ASICs) and field-programmable gate array (FPGA) devices.Implementation of the hardware state machine so as to perform thefunctions described herein will be apparent to persons skilled in therelevant art.

Additionally, the present invention may be conveniently implementedusing one or more conventional general purpose or specialized digitalcomputer, computing device, machine, or microprocessor, including one ormore processors, memory and/or computer readable storage mediaprogrammed according to the teachings of the present disclosure.Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example, and notlimitation. It will be apparent to persons skilled in the relevant artthat various changes in form and detail can be made therein withoutdeparting from the spirit and scope of the invention.

The present invention has been described above with the aid offunctional building blocks illustrating the performance of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have often been arbitrarily defined herein for theconvenience of the description. Alternate boundaries can be defined solong as the specified functions and relationships thereof areappropriately performed. Any such alternate boundaries are thus withinthe scope and spirit of the invention.

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed. Thebreadth and scope of the present invention should not be limited by anyof the above-described exemplary embodiments. Many modifications andvariations will be apparent to the practitioner skilled in the art. Themodifications and variations include any relevant combination of thedisclosed features. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

In the various embodiments described above, unless specifically notedotherwise, disjunctive language such as the phrase “at least one of A,B, or C,” is intended to be understood to mean either A, B, or C, or anycombination thereof (e.g., A, B, and/or C). As such, disjunctivelanguage is not intended to, nor should it be understood to, imply thata given embodiment requires at least one of A, at least one of B, or atleast one of C to each be present.

What is claimed is:
 1. A system for sharing sensor information betweenmultiple vehicles, comprising: an aerial vehicle including a firstcomputing device; a first scanning sensor coupled to the aerial vehicle;a ground vehicle including a second computing device; a second scanningsensor coupled to the ground vehicle; the first computing deviceincluding at least one processor and a scanning manager, the scanningmanager including first instructions which, when executed by theprocessor, cause the scanning manager to: obtain first scanning datafrom the first scanning sensor; and transmit the first scanning data tothe second computing device; and the second computing device includingat least one processor and a detection manager, the detection managerincluding second instructions which, when executed by the processor,cause the detection manager to: receive the first scanning data from thefirst computing device; obtain second scanning data from the secondscanning sensor; identify an overlapping portion of the first scanningdata and the second scanning data based on at least one reference objectin the first scanning data and the second scanning data; and execute anavigation control command based on one or more roadway objectsidentified in the overlapping portion of the first scanning data and thesecond scanning data.
 2. The system of claim 1, wherein the firstinstructions, when executed, further cause the scanning manager to:receive a movement command from the second computing device on theground vehicle, the movement command including a location; cause theaerial vehicle to move to the location; and obtain the first scanningdata at the location.
 3. The system of claim 2, wherein the locationincludes at least one of location coordinates, a position relative tothe ground vehicle or other object.
 4. The system of claim 1, whereinthe at least one reference object includes a representation of at leastone of the one or more roadway objects represented in the first scanningdata and the second scanning data.
 5. The system of claim 1, wherein theone or more roadway objects include a driving lane represented in theoverlapping portion of the first scanning data and the second scanningdata.
 6. The system of claim 5, wherein the second instructions, whenexecuted, further cause the detection manager to: transform the firstscanning data and the second scanning data into an intermediate format;perform object recognition on the intermediate format using a machinelearning model to identify the one or more roadway objects, the machinelearning model trained to identify the one or more roadway objects; anddetermine the navigation control command based on the objectrecognition.
 7. The system of claim 6, wherein an input to the machinelearning model includes a plurality of input patches from theintermediate format, a number of channels, and a height and width of theinput patches, the number of channels representing color and depthassociated with each input patch, and wherein an output of the machinelearning model includes one or more confidence scores for each inputpatch, the one or more confidence scores associated with the one or moreroadway objects.
 8. A method for sharing sensor information in an aerialvehicle environment, comprising: receiving first scanning data from afirst computing device coupled to an aerial vehicle, by a secondcomputing device included in a ground vehicle, wherein the firstscanning data is obtained using a first scanning sensor coupled to theaerial vehicle; obtaining second scanning data from a second scanningsensor coupled to the ground vehicle; identifying an overlapping portionof the first scanning data and the second scanning data based on atleast one reference object in the first scanning data and the secondscanning data; and executing a navigation control command based on oneor more roadway objects identified in the overlapping portion of thefirst scanning data and the second scanning data.
 9. The method of claim8, further comprising: receiving the first scanning data by a pluralityof ground vehicles in communication with the aerial vehicle.
 10. Themethod of claim 8, further comprising: obtaining third scanning data bya plurality of aerial vehicles in communication with the ground vehicle;and transmitting the third scanning data to the ground vehicle.
 11. Themethod of claim 8, wherein the first scanning sensor includes a firstLiDAR sensor and the second scanning sensor includes a second LiDARsensor.
 12. The method of claim 11, wherein the first scanning dataincludes first mapping data generated based on point cloud datacollected by the first scanning sensor and wherein the second scanningdata includes second mapping data generated based on point cloud datacollected by the second scanning sensor.
 13. The method of claim 12,further comprising: combining the first mapping data with the secondmapping data to increase a coverage area of a local map maintained bythe ground vehicle.
 14. The method of claim 12, further comprising:detecting a traffic condition affecting the ground vehicle; and inresponse to the detected traffic condition, sending a command to thefirst computing device on the aerial vehicle to collect the firstmapping data.
 15. A non-transitory computer readable storage mediumincluding instructions stored thereon which, when executed by one ormore processors, cause the one or more processors to: receive firstscanning data from a first computing device coupled to an aerialvehicle, by a second computing device included in a ground vehicle,wherein the first scanning data is obtained using a first scanningsensor coupled to the aerial vehicle; obtain second scanning data from asecond scanning sensor coupled to the ground vehicle; identify anoverlapping portion of the first scanning data and the second scanningdata based on at least one reference object in the first scanning dataand the second scanning data; and execute a navigation control commandbased on one or more roadway objects identified in the overlappingportion of the first scanning data and the second scanning data.
 16. Thenon-transitory computer readable storage medium of claim 15, wherein theat least one reference object includes a representation of at least oneof the one or more roadway object represented in the first scanning dataand the second scanning data.
 17. The non-transitory computer readablestorage medium of claim 15, wherein the one or more roadway objectsinclude a driving lane represented in the overlapping portion of thefirst scanning data and the second scanning data.
 18. The non-transitorycomputer readable storage medium of claim 17, wherein the instructions,when executed, further cause the one or more processors to: transformthe first scanning data and the second scanning data into anintermediate format; perform object recognition on the intermediateformat using a machine learning model to identify the one or moreroadway objects, the machine learning model trained to identify the oneor more roadway objects; and determine the navigation control commandbased on the object recognition.
 19. The non-transitory computerreadable storage medium of claim 18, wherein an input to the machinelearning model includes a plurality of input patches from theintermediate format, a number of channels, and a height and width of theinput patches, the number of channels representing color and depthassociated with each input patch, and wherein an output of the machinelearning model includes one or more confidence scores for each inputpatch, the one or more confidence scores associated with the one or moreroadway objects.
 20. The non-transitory computer readable storage mediumof claim 15, wherein the first scanning sensor includes a first LiDARsensor and the second scanning sensor includes a second LiDAR sensor,wherein the first scanning data includes first mapping data generatedbased on point cloud data collected by the first scanning sensor andwherein the second scanning data includes second mapping data generatedbased on point cloud data collected by the second scanning sensor.